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Abstract 

An elementary proof is given for a theorem showing that certain birth- 
death chains show martingale-hke behavior at large stopping times. This 
is a generalization of and new proof for a theorem from [Marvlj . 

1 Introduction 

Let Xm be a Markov chain taking values on the nonnegative integers with the 
following transition probabilities for n ^ 



(Tn if j = + 1 

/„ if j = n - 1 

if In -j| 7^1. 

Implicit here is the fact that r„ + = 1. We suppose further that Xq = k almost 
surely, for some /c G N, so that at time our chain is at a fixed state. To avoid 
trivialities, assume /n,Tn > for all i. Xm is essentially a random walk on the 
nonnegative integers, moving to the right from state n with probability r„ and to 
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the left with probabihty /„. Such a Markov chain is referred to as a birth-death 
chaii^ This name comes from considering as the number of members in a 
population, where at each step either a new member is born or an old member 
dies, causing the process to increase or decrease by 1. We can assume poo = 1 
and poj = for any j ^ 0, as when the population reaches it is considered 
to have gone extinct with no possibility of regeneration. We let Ta denote the 
first time hits 0, with Ta = oo if it never does. If Z„ = r,„ = 1/2 for all n, 
then our birth-death chain is simple random walk stopped at 0, and is therefore 
a martingale. The well-known Optional Stopping Theorem then implies that 
i?[XT-] = i?[Xo] = k for any stopping time r such that E\r\ < oo. 

Clearly, a general birth-death chain is not a martingale, and therefore -E'[Xt-] = 
£'[Xo] need not hold. However, we can give a sufficient condition such that similar 
behavior holds for large stopping times. The following is the main result of this 
paper. 

Theorem 1 If too := linin »oo exists then liirim ^oo E[Xt^] exists, and 



(2) hm E[XrJ = " 7 "-"-^ 

m—^OD too 

for any non- decreasing sequence of stopping times {Tm}m=i satisfying E[Tm] < oo 
such that Tm. — > Ta almost surely. 

Note that the theorem is valid if too = +oo or 0, with ([2]) being interpreted as 
or oo respectively. The condition -^[Tm] < oo is clearly necessary, since in the 
case P(Ta < oo) = 1 we have E[Xt^] = regardless of the value of too- A 
preliminary version of this theorem has already been proved in |Marvl] : only the 
case Tjn = m A Ta was considered, but the extension to arbitrary stopping times 
causes no real difficulty, as will be shown below. The proof given in |Marvl] 
relies on properties of Brownian motion and its corresponding theory of local 
time. Given the simplicity of the statement of the theorem, it seems desirable 
to have a proof which is elementary and self-contained, without reference to the 
machinery of Ito calculus and the like. The purpose of this paper is to give such a 
proof, as well as the extension to more general stopping times than = m ATa. 

^Onc may also find the name state dependent random walk in tlie literature. 
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2 Proof of Theorem 



If we condition on Xm'-i = n, then if we move to the right we increase the 
expectation by 1, and if we move to the left we decrease by 1. It follows that 

oo 

(3) E[Xm'] - E[Xm'^i] = 5^P(X„_i = n)(r„ - /„) 

n=l 

Summing over all < m' < m gives 

oo 

(4) E[X^] -k = E[X„,] - E[Xo] = E[Gl] (r„ - /„) 

n=l 

where is the number of times that X is equal to n on or before time m — 1. 
We need the analogous formula for stopping times. 

Lemma 1 Let T be a stopping time with E[T] < oo. Then 

oo 

(5) E[XT]=k + J2E[G^]irn-ln) 

n=l 

where G^ is the number of times that X is equal to n on or before time T — 1 . 
Proof: For any integer h>{] lei Sh = T /\h. For /i' > 1 we have 

oo 

(6) E[Xs^, - Xs^,J = Y,nT>h'- 1, Xh'-i = n)(r„ - Q 

n=l 

Sum this expression over 1 < h' < h and use ^'[Xgo] = k to get 

h oo 

(7) E[Xs,] = + ^ 5^ P(T > /i' - 1, = n)(r„ - /„) 

h'=l n=l 

As Sh < h we have supo<m<5^ X^ < k + h, and {T > h' — 1} = {Sh > h' — 1}. 
Thus, the following manipulations are justified. 
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h k+h 



E[Xs,] = k+J2Yl ^("^'^ >h'-l, Xh'-i = n)(r„ - Q 

hi=l n=l 

k+h h 

= k + 5^(r„, - /„) ^ P{Sn >h'-l, X,,_i = n) 

n=l h'=l 
k+h h 

= k + Y^i^n - QElY, mSh >h'- = n})] 

n=l h'=l 
k+h 

= k + Y,E[Gl]irn-ln) 



71=1 

where 1{A) denotes the indicator of the set A. Since E[T] < oo, hm/j ^oo E[T — 

Sh] = 0, and since |Xm+i — Xm\ = 1 almost surely we have — XsJ] < 

E[T — Sh] — y as well. The result therefore follows by letting h — > oo in ([8]). 
n 

Let to 1 and 

(9) := 

rira . . . r„ 

for n > 0. The following lemma gives the exit probabilities of an interval. 

Lemma 2 Suppose Ym is a birth-death chain with the same transition probabili- 
ties as Xm but with Yq = k' almost surely. Let 0<a<k'<b<oo, and let r be 
the first time that Y^ hits a or b. Then 



(10) P{Y^ = a) = ^^, Piyr = b) = ^dj^ 

There are a number of ways to prove this well-known fact. One way is to write 
down the correct recurrence relations and verify that the given expressions satisfy 
them; see |Nor98] or |Sys92| for the details. Another is to use the relationship 
between electrical resistance on graphs and random walks; see |DS84j for an 
elegant introduction to this technique. Yet a third is to realize the birth-death 
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chain as a Brownian motion evaluated at a properly chosen sequence of increasing 
stopping times; see |Marvlj . Setting a = and letting b — y oo we obtain 

(11) P{X^ = for some m) = ^ST"" 

with the quotient interpreted as equal to 1 if the sums diverge. Note that if 
P{Xm = for some m) < 1 then the sums in (fTTj) converge so that tn — as 
n — > oo. Furthermore, in that case we must have — > oo on some set of 
positive measure, so that E[Xt^] — > oo, and we see that holds trivially. We 
will therefore assume from now on that P{Xm = for some m) = 1. In what 
follows, let Xn = Y17=o^i- Lemma [1] shows the importance of calculating E[G^^], 
and we therefore need the following. 

Lemma 3 

mm{xn, Xk) 



(12) E[G-^]= hm 



tn—lln 



Proof: That E[G^^] = lim^ >oo-E'[G'^^] is clear by monotone convergence. Let 

jn = E[GJ^^], and let us first suppose n = k. U Xi = k + 1, then X must 
eventually return to k, whereas ii Xi = k — 1 then by Lemma |2] the probability 
of returning to k before hitting is Thus, 7^ satisfies 



(13) 'yk = I + Tk'yk + lk{^ 

Solving for 7^ gives 



Xk~i 

Xk 



Ik 



1 Xf^ Xfc Xk 

Ik 



l-rk-lk(^ 



a;fc(l - Tk) - Xk-ih {xk - Xk-i)lk tk-ik 



For general n, we let Ej[G^^] denote the expected number of times that the 
birth-death chain hits n before hitting 0, where has the same transition 
probabilities as X^ but Yq = j almost surely. Then by the strong Markov 
property of X we have 

(14) E[G^J = P{Xm hits n before 0)E4G^J = P(X„ hits n before 0) ^" 



tn—l^ri 
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The general result now follows by noting that P{Xm hits n before 0) is 1 if n < /c 
and — ii n > k hj Lemma [21 □ 

We see initially from this that E[G^^] < ^^yT' Returning to (j4]), we may write 

{rn - Q = tn-iln{j^ - j^) ■ If we assume Y^'^=i li^ " tZ:^\ < then by (SD we 
have 



(15) 



n=l 

oo 



n=l 



tn—lln 



tn tn—l 



71=1 



< OO 



We may then apply the dominated convergence theorem(to the functions fm{n) = 

El'GTnM^n — In) defined on N) to conclude that lim,^ yooElXq-Jl exists and is 

given by 



hm E[XTj = k + Y,E[GT.]K~ln) 



m >oo 

n=l 



k oo 

(16) "r . , 



n=l '^'^ ^"-1 n=k+l " '^-l 



fc-1 



1 1^1. , , 1 1 ^ 

= k + Xk— - - 2^ —[Xn - Xn-l) + Xk[- — ) 

where summation by parts(see |Lan97j ) was used in passing to the last line. It 
was also assumed that Q < < oo, but the cases too = 0, oo are handled easily 
by the same technique. As — = = 1, this simplifies to 

1 + + . . . + hiilh^ 
(17) hm E[XrJ = ^ = 

m^oo too too 

which is the desired expression. Now we must remove the assumption Xl^i l*'^ — 
< oo. We need the following lemma. 

Tn—l 
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Lemma 4 If t is any stopping time, then E[G'^]tn-iln > E[G^~^^]tnln+i for any 
n > k. 

Proof: Using the strong Markov property of Xm and Lemma [3], we have 



(18) 

E[G'J:]tn-lln = E[G^^]tn-lln — E[ ^ (Xm)]tn-l/n 

m=T+l 

oo 

tr..,Lj2PiXr+l=j)E,[G^J 

j=l 
n 

tn-l^n P{^T+1 = j)- —J tn-iln P^Xt^i = j 



j=l j=n+l 



n oo 
j=l j=n+l 



Likewise, 



n+l cxD 

(19) = - ^ P(X.+i = j)x, - ^ P(X.+i = j)xn+i 

j=l j=n+2 

The result now follows from f ITH]) . ffUJl) . and the fact that > □ 

Let us suppose initially that for each m we can find a number Nm > k such that 
supo<m'<T„ ^m' < almost surely. Returning to @, we have 



(20) E[XrJ = k + Y,E[GJi.Jt^^M--^)+ E ^[GLl^n-i^nl- ^ , . 

n=l 7i=/c+l 

since E[G^^] = for > A''^. As -^[(7^^] — > E[G^^], the first sum converges 
to 
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k ^ ^ 

^ ElG^Jin-llni ) 
Zfi i-n — 1 

n=l 


k 

= ^Xni 

n=l 


. 1 

f 


/) 


(21) 




1 

= Xfc 

tk 

1 

= Xk 

tk 


1 

xi- 
to 

k 


— ^ — (a^n+i — a:„) 

n=l ^'^ 



We therefore need only show that the second sum converges to the proper thing. 
Let us assume that too ^ (0, oo), and let e > be given. By summation by parts, 
using E[G^^^^] = 0, we have 
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(22) Yl E[GTjtn-M- - ^) = 

n=k+l ^'^ ^""-^ 

1 ^"^ 1 

~ — E[G!^^^]tklk+l+ ^ —{E[G^^]tn~iln - E[G'!^^^]tnln+l) 

^k , , , f-Ti 

n=fe+l 

Note that Lemma H] implies that the terms in the sum on the right side of ( l22i) are 
all positive. We may choose N > k such that j- G {-^ — e,j- + e) for all n > N. 
Having chosen this, we may choose M > N — k such that Xk > E[G^^]tn-iln > 
Xk — e for all n G [A;, + 1], m > M. Setting = sup^-^o f- and using (!22l) . we 
see that for m > M 
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(23) E[G-Tjtn-M- - ^) 

n=k+l 



< 



1 ^ 1* 



n=k 



< -U^k -e) + \\E[G'^+^]t,h^, - 

tk t 

+(t \- ^){E[G^^^]tNlN+l - E[G^2^^]tNjN^+l] 

too 

< -— (a^fc - ^) + 7 ^ + + ^)^fc ) 
As e > is arbitrary, this shows that 

(24) hmsup J2 E[G-Jt„.M- " ^) < " -) 

Proceeding similarly, we can obtain 



111 

(25) hminf V E[G^^]t„_i/„(- - ^) > Xfc(- 

We conclude that 



1 



n=fc+l 



) = Xk{- -) 

Combining this with fl2T|) and f l20|) gives the desired result. We assumed G 
(0, oo), but the cases -r- G |0, ooj are similar but easier and are omitted. At this 
point we have but a single assumption which remains to be removed, which is 



(26) 



Nm ^ 

hm Yl E[G-rm]ir.-M- 



n=k+l 
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that for each m we can find a number Nm > k such that suPq<„/<7.^ Xm' < Nm 
almost surely. Suppose now that is a sequence of stopping times as in the 
statement of the theorem. As E\Tm\ < oo, there is a number Hm such that if 
Sra = A Hm, then E[Tm - 5"^] < ^. Since \Xm+i - X^l = 1 almost surely, we 
see that \E[Xt„,]- E[XsJ\ < E[Tm-Sm] < ^- Since supo<^/<5„ X^' < k + Hm, 

we see by our previous work that lim^ ^oo E[Xs^] = f^, and it follows that the 

same is true for lim^ ^oo E[Xt^]. This completes the proof of the theorem. 



3 Examples 

Example 1: This example appeared in |Marvl] . Let = 2^^^! ''^n = for ^ ^ 
1. Then t„ = so that too = 0. On the other hand, x^o = 1 + tn = oo. 

We see that the birth-death chain Xm built upon these transition probabilities 
has an extinction probability of 1, but — > oo as m — y oo. 

Example 2: Suppose there is an M > 1 such that t„ = r„ = 1/2 for all n > M. 
Then t^o = , and so for a sequence {Tm}m=i satisfying the requirements for 
our theorem we have 



Ll A_ _|_ ^l---^fc-l 

ri ■ ■ ■ ri...rfc_i 



(27) E[XtJ ■ 

ri...rM 

Essentially, if Xm reaches large n it performs simple random walk, and for that 
reason it is not a surprise that it behaves like a martingale for large stopping 
times. 

Example 3: In order for too to exist and lie in (0,oo), it is clearly necessary 
that VnJn — ^ 1/2 as — > oo, but is not sufficient. Example 1 shows this, as 
^oo = 0. Exchanging r„ and In in Example 1 will give the case when t^o = oo. 
It is not hard to interlace these series in a way so that too does not exist. The 
theory of infinite products(most complex analysis texts contain this, for example 
see [Ahl78j ) gives us the standard necessary condition for too to exist and lie in 
(0, oo). The following proposition is obtained. 

Proposition 1 Suppose that 



oo , 

(28) 



n=l 



< OO 
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Then linim ^^E[Xj-^] exists and lies in (0,oo), where {Tm}m=i '^'^?/ non- 
decreasing sequence of stopping times satisfying E[Tm] < oo such that — > 7a 
almost surely. 



4 Concluding remarks 

The reader patient enough to study the proof given here as well as the one given 
in |Marvl] will no doubt recognize that each is simply the other in disguise. As 
alluded to in the introduction, the one given here has the advantage of being free 
of stochastic calculus, which may appeal to the sensibilities of some readers. On 
the other hand, the other argument shows the local time of Brownian motion to be 
a very useful tool for bookkeeping, and it is doubtful that the author would have 
noticed this theorem without studying the Brownian motion model. Regardless, 
it would be wonderful to have a truly different proof, for instance using electric 
resistance or perhaps a cleverly constructed change of measure. 
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